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Measuring and Validating the Characteristics 
of Instructionally Effective Schools 

Introduction 

The purpose of this paper is to describe efforts under- 
taken as a part of the Connecticut School Effectiveness 
Project to collect meaningful, valid and reliable data on 
the characteristics of instructionally effective schools. 
Data collection instruments and procedures have been 
designed to both validate the constructs upon which the 
r Connecticut model (Gauthier, 1982) is based and to provide 

information for school improvement. 

The characteristics that describe instructionally 
effective schools have been derived from the emerging 
body of research on school-based variables that have been 
shown to be consistent correlates of substantial student 
achievement. These characteristics include a safe and 
orderly climate, a common sense of purpose <£r mission, 
s'trong instructional leadership, high expectations and a 
sense of efficacy, a substantial amOurit^oT^Alme spent on 
focused teaching and learning, and purposeful parent and 
community involvement. (Brookover and Lezotte, 1977; 
Clark, Lotto and McCarthy, 1980; Comer, 1980; Edmonds, 1979; 
Edmonds and Fredericksen, 1978; Madden, Lawson and Sweet, 
1976; Mann, 1980; Rutter et al . , 1979; State of New York, 
1974; Weber, 1971.) 
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The research that these characteristics have been drawn 
from is primarily descriptive, qualitative and correlational. 
As with much of the research in education there has been no 
attempt to link the findings to an established theoretical 
framework. The characteristics have generally been outlined 
as descriptive, behavioral and procedural patterns. Research 
efforts and school improvement projects refer to these , loosely 
coupled characteristics as empirically established variables 
or constructs. It is imperative to operationally define 
each characteristic, identify underlying constructs and link 
t v ie constructs together in a theoretical framework; Effdrts 
in this area in the Connecticut Project are designed to extend 
and develop the present research base and to establish a 
solid foundation on which to build focused school improvement 
projects. 
Background 

In the spring and fall of 1980 the Connecticut State De- 
partment of Education, along with a number of regional 
educational centers and local districts, began to explore 
processes and procedures for the development of a model for 
instituting and supporting comprehensive, school-based improve- 
ment efforts focused on improving student achievement. The 
characteristics of effective schools, previously referred to, 
were the foundation of the planning efforts. An application 
and extension of the school effectiveness research encompassing 
instruction, curriculum^ organizational dynamics, change theory 
and community involvement was the primary goal. 
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A process was developed that advocates a voluntary, 
school-based approach that helps the school examine itself 
introspectively in relation to school effectiveness charac- 
teristics and develop and implement action plans that are 
meaningful to the principal and faculty of that" school. 
(Gauthier, 1982) 

During the 1980-31 school year approximately ten 
Connecticut schools volunteered to enter into the Connecticut 
School Effectiveness Project. Data on the presence of each 
of the school effectiveness characteristics were collected 
by an % assessment team using 1 interview and questionnaire 
schedules developed by the staff of the New York City School 
Improvement Project (1980). Achievement data from each of 
the schools were also collected and portrayed for the school 
principal and faculty. Assessment teams were trained in 
use of the instruments by personnel from the New York City 
Project. Achievement data was analyzed by Connecticut State 
Department of Education personnel. In all cases, data were 
gathered and portrayed by trained assessment teams to enable 
principals and teachers to make Judgements and decisions about 
their own schools. ^ 

School teams analyzed their own data and with the assist- 
ance of a trained consultant, developed action plans focused 
on school effectiveness characteristics that they determined 
to be of highest priority. These plans are presently being 
implemented according to the time line's established by the 
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school teams. Shoemaker (1982) has presented an analysis of 
evaluation- strategies' for these efforts. 
Rationale for Assessment Revisions 

, Although improvement efforts with this first group of 
schools have proceeded satisfactorily certain aspects of the 
assessment were identified as particularly problematic by 
the assessment team members, the faculties and principals of 
participating schools and project coordinators. Based on 
this experience, the Department of Education staff planned 
significant revisions to the assessment phase of the project. 
The revised process was planned with two«primary goals: 

1. To provide methods fof gathering and portraying reliable, 

comprehensive and integrated information to be used 

in school-based improvement efforts, 
\ 2. To gather data in a manner that will further define 

alterable, school level variables that effect student 

achievement . 

The New York City School Improvement Project Instruments , 
Connecticut is indebted to the staff from the New York *<*ity 
School Improvement Project for permitting their instruments 
to be used to assess Connecticut schools during the first 
of the project in Connecticut. Despite the valuable and use 
information gathered using the New York project instruments 
it was perceived that more, and in some cases different, infor- 
mation was needed for the school improvement process and 
research efforts developing in Connecticut. Interview and 

4 
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questionnaire items did not adequately reflect the school 
effectiveness characteristics as they vfere operationally 
defined ia the Connecticut model. 

The open ended nature of the interview questions caused 
reliability among raters to be questioned. The information 
collected was in many cases a function of the extent and 
nature of the interviewees probes. Although interviewers 
did receive training, the expertise required to gather infor- 
mation In this relatively unstructured manner was significant • 
A method to code and categorize narrative responses without 
adversely effecting the accuracy of the information was 
determined to be needed. 

Procedures used for data summary were developed during 
the first year. There was not a systematic way to synthesize 
or summarize the vast amount of recorded Information result- 
ing from the interviews. Attempts were made to generalize 
concerns and areas to be looked at from pooled responses. 
A reliable procedure to guide this complex data summary did 
not exist. 

Similarly, data collected from the questionnaire was 
summarized in a number of ways during this initial year. The 
format and content -of the questionnaire was perceived to 
be in need of significant redevelopment to meet the goals 
identified for the Connecticut project. 

Determination of Appropriate Instruments and Procedures . 

As the revised assessment process was planned it was 
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acknowledged that a comprehensive assessment process is needed 
to reflect the complexities of the social system of a school 
organization. Multiple methodologies as well as multiple 
data sources are crucial in developing this comprehensive 
view (Sirotnik and Oakes, 1981). 

Given the positive experiences with the New York project 
interview, the descriptive correlational research that Jthe 
school effectiveness characteristics have been drawn from, 
and the lack of a clear understanding of how these characteris- 
tics bring about the if effects, the focused interview was 
determined to be the. primary data collection tool in the 
Connecticut model. In an area that requires extensive explora- 
tory research and planned theory development, the interview 
has been identified as a key device to obtain research - 
relevant information focused on content specified by research 
objectives of systematic description and explanation (Cannell 
and Kahn, 1968) . Given the stated goals of the Connecticut 
project the development of a structured interview schedule 
specifically focused on the operationally defined effective 
schools characteristics was necessary. 

A paper and pencil technique designed to measure the 
same characteristics as the interview was also agreed upon. 
The questionnaire format was to be developed in a manner that 
would facilitate integration with the interview data. In 
the attempt to validate the identified constructs different 
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methods of measurement should converge on the construct 
(Kerlinger, 1973 ) . The questionnaire was to be developed 
to validate and corroborate the data collected through the 
interviews. The fact that the entire faculty may not be 
interviewed was also considered in deciding on the use of a 
questionnaire- Faculty members not included in the interview 
would be asked to complete a questionnaire. This would ensure 
that the summarized data included information from an entire 
school faculty. 

The questionnaire and interview schedules were to be 
developed to measure perceptions in relation to the existence 
of the seven school effectiveness characteristics. Experience 
with previous assessments, al'ong with the commitment to multiple 
data sources, lead to the consideration of gathering additional 
information to supplement, reinforce and support data from , 
the interview and questionnaire. Qualitative research procedures 
were considered (Bogden and Taylor, 1975). It was decided that 
archival data in the form of documents, written reports, records, 
policies, etc. would be helpful. Archival data generally include 
information which is readily available and which do not require 
a formal data collection instrument. Perhaps the greatest 
advantage of these data is that the assessment procedures do 
net themselves influence the results. Archival data could 
^rovide and serve well as a reliability, check for the other 
observations collected. These data would also help the school 
improvement team gain a more complete picture of the organization. 
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A method and format for portraying school achievement 
data were also determined to be needed. Data from schools 
was found to be collected and reported back in many formats. 
Key decisions at the core of the Connecticut School Effective- 
ness Project were based on patterns of student achievement 
in basic skill areas. Methods used in the past were not 
systematic or flexible enough to provide useful information 
for schools. A format was needed to portray school achieve- 
ment data in a manner that would allow decisions to be made 
by a school faculty and principal based on Edmond's (1979) 
definition (and subsequently Connecticut's) of an effective- 
school . 

School Improvement Considerations . The assessment 
process would collect data from the sources listed above. 
Methods were to be developed to portray the summarized, 
collected data in a meaningful, integrated manner. The 
combination of data sources and methodologies would provide 
the school planning team an objective, comprehensive picture 
of their school in relation to the existence of the effective 
school characteristics and in relation to patterns of student 
achievement portrayed as the Connecticut model outlines. 
Informed, focused action planning could be based on these data. 

Research: Validation of the Characteristics . One of the 
first steps required to design more controlled studies of 
"school effect" is the identification of the constructs under- 
lying the descriptive characteristics derived from the 
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descriptive research on effective schools (Edmonds, 1979; 
Brookover and Lezotrte, 1979). Michael Cohen (1920) has 
strongly suggested further methodological work to more 
adequately operationalize the constructs.- The Connecticut 
School Effectiveness Project has identified the development 
of this research as one of its primary goals. 

Carefully designed instruments high in content validity 
can be used to further develop the construct validity of 
the operationally defined characteristics being measured. 
The possibility of developing a multitrait-multimethod 
matrix (Campbell and Fiske, 1959) from questionnaire and 
interview data can also be used to validate the constructs. 

The assessment processes has been planned and designed 
to address the need for improved and additional research 
on the school effects characteristics. Assessment and school 
improvement efforts in Connecticut continue with Dale Mann's 
statement in mind, ..."if the prospect for better research 
obviated attention to existing research there wpuld be no 
reason to read any current research" (Mann, 1980, p. 8). 
Data Sources and Instruments 

The Connecticut School Effectiveness Assessment Process, 
therefore, consists of the following components: 

1. The Connecticut School Effectiveness Interview : A 
structured interview schedule with a modified fixed- 
alternative item format designed to assess the degree 
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of existence of seven characteristics of school 
effectiveness identified in the Connecticut model. 
(See Appendix A for a sample) * 
2. The Connecticut School Effectiveness Questionnaire . 
A paper and pencil technique to assess the degree 
of existence of seven school effectiveness 



characteristics identified in the Connecticut 
model. The 100 item instrument is designed to 
parallel the Interview. (See Appendix B for a 
sample) 

3. Archival IJata Profile: A format for analysis of 
available documents to support and supplement the 
data collected with the interview and Questionnaire. 
Documents include but are not limited to reports, 
written policies, records, memorandum, etc. 

4. Achievement Data Profile: An analysis of student 
achievement scores which illustrates similarities 
and differences afcong students along social class 
dimensions. 

The following sections will describe the initial procedures 
used to develop the foregoing assessment components. 
DeveloDifient of Asses sment Components 

The Connecticut School Effectiveness Interview . In order 
to develop an interview schedule that reflected the defini- 
tions of the school effectiveness characteristics identified 
in the Connecticut model ft was necessary to thoroughly 
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review the school and classroom effectiveness literature 
'-and identify behavioral indicators of the properties to be 
measured. This procedure established a degree of content 
validity for the interview items.' The analysis also resulted • 
in operationally defined characteristics that can be measured 
through properly designed instruments. The measurement of 
each characteristic was then to be used to define the 
constructs (Kerlinger, 1973). 

Instruments from studies that identified instructionally 
effective schools were also analyzed (School Improvement 
Project Needs Assessment Instruments, 1980; L. Lezotte's 
interview schedule used in Lansing, Michigan). 

A pool of potential interview items was developed for 
each characteristic. The immediate goal was to develop 
interview items which would adequately represent the dimensions 
of each characteristic. As items were developed the process 
was guided by Kerlinger' s question, "Is the substance or 
content of this measure representative of the content or 
universe of content of the property being measured?" 
(Kerlinger, 1973, p. 458). Kerlinger's recommendations on 
question-writing in developing interview schedules were also 
carefully considered. The interview wa^ to be used both as 
a research tool and as an instrument to collect useful accurate 
data for school decision-making. These dual, but related, goals 
were continually balanced. 

% 

As an indicator of content validity a panel of experts, 
State Department of Education and local district personnel 
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thoroughly familiar with school and classroom effectiveness 
research, were asked to sort the randomly ordered items into 
appropriate operationally defined categories. The panel also 
made recommendations as to whether or not each item would re- 
main or be modified. 

Items were then grouped by characteristic and piloted 
on teachers and administrators to improve item clarity and 
to elicit a sampling of responses. These procedures resulted 
in a first draft consisting of 63 item stems (August 1981). 

Interview Format and Coding . Experience with the New 
York Needs Assessment as previously described lead to the 
design of a format that would combine the strengths of a 
standardized format and a modified fixed-alternative coding 
scale. This format was used to decrease bias and low 
reliability among interviewers and to increase the accuracy 
of the summarized information portrayed for the school. 

Descriptive continua were developed for each item. Each 
item has five descriptive phrases representing the range of 
responses that might be possible for each item. The descrip- 
tors are arranged in a left to right sequence with the far 
right descriptor the research-based, optimal response 
(Appendix A). The descriptors are designed for use by the 
interviewer s not to be shared with the respondent. The 
Interview schedule is structured, the respondent responds 
to open-ended questions and the interviewer has a fixed- 
alternative scale to code the response. Interviewers receive 
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specific training in recording information from the interview. 
An understanding of the construct being measured is essential. 
The interviewer listens for key words and categorizes the 
response along the continuum. A comment section included in 
each section of the interview is used to record any, and all, 
re]3vant information that is not represented along the 
continuum. 

Interviewers are trained to use controlled non-directive 
probing including encouragements, silences and appropriate 
interruptions (Richardson et al.> 1965; Cannell and Kahn, 
1968). 

The result is a classification of responses in one of 
five categories. Each category is assigned a numerical code 
and responses are able to be summarized in a format useful ' 
for exploring research questions and accurately portraying 
data for school decision-making. Responses are coded on an 
answer sheet under columns marked 1-2-3-4-5 next to abbreviated 
item stems. 

Try out Procedures . School district administrators 
and teachers were contacted and asked to nominate schools 
that they perceived to be either exceptionally ineffective or 
exceptionally effective in relation to their success in 
producing patterns of high achievement for all students. ' 
Six teachers in five Connecticut public schools were subse- 
quently interviewed. This modified "known groups" procedure 
did produce meaningful and significant variance among the 
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schools represented. Due to the extremely small sample and 
different interviewers, this procedure provided some informa- 
tion to plan a more psychometrically sound pilot, but not 
enough information to produce a sophisticated statistical 
analysis. Information from each of the interviews was 
thoroughly reviewed and numerous modifications in item stems 
were made. 

The result was an Interview schedule revised and ready 
for piloting. Field-testing/indicated that a complete inter- 
view would take from *J5 minutes to one hour. Three open-ended 
questions were included at the end of the interview to give 
the respondent a chance to mention any area not covered in 
the interview. Interviewers were to record all relevant 
information offered in response to these questions. 

The Connecticut School Effectiveness Questionnaire . The 
Questionnaire was developed as an additional data collection 
method to validate and corroborate information collected 
through interviews. Items were developed from the pool of 
behaviors and activities used in developing the Interview. 

The content represented within the groups of questionnaire 
items was matched with the Interview. Content validity was 
established through agreement by a panel of Judges concerning 
the appropriateness of each item as being representative of 
each of the operationally defined constructs. Also, a card- 
sort procedure was used by judges to sort items into appropriate 
categories. Based upon these results items were modified or 
deleted. 
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The Questionnaire that was developed had 100 items. 
The items are not grouped by characteristic but are placed 
randomly throughout the instrument .Vjksponses may be 
summarized by characteristic by referring to a coding chart. 
The response scale was designed in a Likert format: SD - 
Strongly Disagree; D - Disagree; U - Undecided ;^ A - Agree; 
SA - Strongly Agree. The potential. of response set is reduced 
by randomly positioning the items and by constructing every 
sixth item to be reversed scored. (See Appendix B for a sample) 

The Likert scale was used so that data could be accurately 
and efficiently summarized and portrayed for schools. The 
scale also permits a numerical cede to be assigned as a 
means of quantifying the data. Data in this form would be 
able to be integrated with the Interview data in a reliable, 
systematic manner. 

The Questionnaire was tried out with six teachers. 
Teachers were asked to comment on the clarity of the items, 
the clarity of the directions and the general affect created 
by the instrument. y 

The result was a questionnaire ready for field testing. 
Try-out procedures indicated it would take approximately 
30 minutes to complete the questionnaire. The schedule was 
designed to need minimum administration direction. It was 
to be administered at a brief faculty meeting following the 
days on which interviews were held. 

Archival Data . The procedures and format developed for* 
collection of documents and records were simply a checkliit 
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of possible materials to be collected. The materials were 
identified with one of the seven effectiveness characteristics. 
The list provided below is made up of - examples of archival 
data. Other information is added based on the needs and 
requirements of the assessment team and school faculty. 
SAFE AND ORDERLY ENVIRONMENT Discipline Policy * 

Infraction Data 

Vandalism Data 
CLEAR SCHOOL MISSION Statement of Purpose 

Written Objectives 

Mastery Requirements 
INSTRUCTIONAL LEADERSHIP Formal Observation Format 

Instructional Planning Guidelines 

Staff Development Program 
HIGH EXPECTATIONS Promotion and Retention Policy 

Retention Data 

Grouping Practices 
TIME ON TASK Allocated Times for Instruction 

Attendance Data 

Homework Policy 

Library Usage 

MONITORING STUDENT PROGRESS Standardized Testing Plan 

Test Analyses 
Report Card 

Other Systematic Reports of 
Student Progress 
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HOME SCHOOL RELATIONS PTO Membership 

Open House Participation 
Regular Newsletter 
Other Parental Communication 
Student Achievement Profile . An important part of the 
assessment process is an analysis of student achievement 
scores. The definition of an instructionally effective school 
used in this project is clear. A school is instructionally 
effective when the proportion of low-income children attain- 
ing at least minimum mastery of basic skills is the same as 
the proportion of all other children obtaining mastery and a 
high percentage of all children are mastering basic skills 
(Gauthier, 1982). 

The analysis necessary to measure effectiveness based 
on this definition required a measure of student achievement 
in basic skills, an index- of social class for each student, 
and a criterion designation for minimum mastery. Shoemaker 
(1982) has developed a format and procedures that facilitates 
a flexible yet systematic data presentation encompassing all 
of the requirements listed. (See Appendix C) 

The sample profile in Appendix C is a typical pattern 
of achievement scores in schools profiled to date. Grade 6 
reading scores on the California Achievement Test are shown. 
Percentile scores (1-99) are listed across the bottom. 
Students are divided into two general categories: Low-Income, 
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and Middle Income. Each (X) represents the score for one student. The 
figure shows that a high concentration of low income students (b0%) 
are clustered below the 30*ile. On the other hand only 20% of the 
middle income students fall below this pre-selected criterion score. 

An instructionally effective school would produce a pattern 
of data in which very few students fell below the criterion 
score and proportions of both low income and other students 
that did would be approximately equal. 

This format for portraying achievement data permits 
school planning teams to identify improvement goals with 
considerable specificity. Action plans designed to improve 
or enhance the highly interactive school effectiveness 
characteristics are ultimately focused on improved student 
achievement . 

Readers interested in more derailed information concerning 
the achievement profile are referred to Shoemaker (1982). 
Initial Pilot of the Assessment Process 

The redesigned assessment process was piloted in one 
Connecticut elementary school in October 1981. A team of three 
trained assessors from the State Department of Education 
administered the Questionnaire and the Interview to all 
teachers and the principal over a three-day period. Each 
respondent was given an identification code for the Interview 
and the Questionnaire so that summarized response patterns 
could be compared. * 



21 



19 

A meeting was arranged with the principal to discuss the 
collection of archival data. The availability and potential 
usefulness of a suggested list of data sources were reviewed. 
A final list was developed and archival data was collected 
and analyzed. 

Information was thoroughly analyzed by the assessment 
team and the School Effectiveness Project staff. Numerous 
procedures and formats for portraying the data were considered 
and developed. Response frequency profiles for each characteris- 
tic and for each item on the Interview and Questionnaire were 
developed. Data patterns from the Questionnaire were developed. 
Data patterns from the Questionnaire were compared with 
patterns emerging from the Interviews. Various statistical 
procedures to summarize and analyze the data were attempted. 

Data were presented to the faculty and principal in a 
number of formats for decision-making. The faculty and 
principal were aske.d to comment on the formats and procedures 
found to be most useful. 

Items that did not perform well on the Questionnaire and 
Interview were identified for possible revision. 

The results of this pilot were used to improve both the 
assessment process and the instruments. Furthermore, the 
school faculty reported that they received meaningful, useful 
data for action planning in relation to the effective schools 
characteristics . 
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Training; For Assessment 

I n November, the Connecticut School Effectiveness 
Assessment Process was reviewed in detail by an invited group 
of State Department of Education, local school district and 
higher education officials. The goals of the session were 
to present the entire process for discussion rnd review, to 
train potential assessors to use and score the Interview 
and to gather data for further instrument development. 

The literature describing the use of a focused interview 
for research purposes is quite explicit in regard to the need 
for thorough training on use of the interview schedule to 
increase reliability (Cannell and Kahn, 1968; Maccoby and 
^ Maccoby, 195^5 Richardson, et si. 1965). 

Training for the Interview consisted first of a detailed 
review of guidelines for interviewing and coding responses. 
Participants familiarized themselves with the Interview form, 
the Answer/Coding Sheet and the general format of the items 
and descriptors. Each effective school characteristic was 
described and operationally defined. After each characteris- 
tic was presented and the items representing that characteristic 
reviewed, a training tape of an actual interview was played. 
A representative sample of items from each characteristic 
was presented on the tape. The tape was stopped after each 
item and the trainer identified the appropriate descriptor 
to be coded. Participants were encouraged to discuss responses 
which were not clear or about which there was disagreement. 
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As the final activity in the training session, participants 
coded a complete interview. The training tape, used as the 
mastery check, required coding skills of various levels of 
difficulty. The tape included examples of probing techniques, 
paraphrasing, and appropriate paci j. 

At the end of the training session a mastery tape of a 
pre-recorded interview was played for participants to score. 
Upon completion of the exercise the coding judged to be most 
appropriate was distributed. 

Throughout the training session questions, comments and 
suggestions were recorded for future consideration. Comments 
indicated that the assessment process and instruments were 
favorably received. 

Further Refinements of the Interview and Questionnaire 

Data collected from the training session was also planned 
to be used to further refine the interview items. Twenty-six 
participants had completed the coding of the mastery tape 
during the training session. These data were analyzed in 
a number of ways to evaluate the effectiveness of the training 
and to provide information for further item and descriptor 
revisions. 

If the criterion score for mastery were set at 80?, 
thirteen participants (50%) would have achieved mastery on 
coding the final interview. Table 1 indicates the inter - 
rater agreement percentages for each item. Items 3,20,21, 
and 22, for example, had less than twenty percent of the 
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participants coding the response in the category judged to 
be most appropriate. A number of explanations for the results 
needed to be explored. Table 1 provided a rich source of 
data for analysis. 

Table 2 provides further information regarding the 
reliability of the instrument. For each scale (representing 
a characteristic) the estimated reliability for one rater 
was calculated from the data obtained from 26 raters (Guilford, 
1953). Some of the scales were associated with relatively 
low reliability estimates and were in need of further revision. 

NEREX Recommendations. Shortly after the November training 
session a proposal was developed by Connecticut State Depart- 
ment of Ed^ation personnel and accepted by New England 
Regional Education Exchange to further develop the assessment 
process. Dr. Robert Gable from the University of Connecticut 
and Dr. Jennifer Greene from the University of Rhode Island 
reviewed the entire assessment process. Specific recommenda- 
tions in the area of psychometric improvements and analyses 
methods for data portrayal were developed. Recommendations 
and suggested revisions were includei in the on-going process 
of instrument development (NEREX, 1981). 

Refineme nt Procedures . Based on the November inter-rater 
agreement percentages, a thorough review of each Interview 
item and descriptive continuum, a review of a transcript of 
the mastery tape and recommendations from the NEREX Report 
numerous revisions were made in the descriptive continua. 



Table 1 



Inter-Rater Agreement Percentages 
For the School Effectiveness Interview 3 
(N«26 Raters) 
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* Ratings 
Scale Item 1 2 3 
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4 




5 






31 


65 


4 




6 




89 


1 1 








7 






8 


(42) 


50 




8 




4 


81 


" 11 ■ 


4 




9 


12 


65 


19 


4 
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Scale 



Item 



Ratings 
2 3 
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*Note that underlined percentages reflect State Department reconaended answers. 
Numbers in parentheses are also acceptable answers. 
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Table 2 



Inter-Rater Reliabilities 
For the School Effectiveness Interview 3 







Number 
of Items 


Estimated 
Reliabilities 
for 1 Rater 


I 


Safe and Orderly Environment 


5 


.587 


T T 
1 1 


Liear ocnooi Mission 


1 1 


• O JO 


T T T 

III 


Instructional Leadership 


1 A 

14 


CQC 


IV 


High Expectations 


10 


.542 


11 

V 


Opportunity to Learn and Student 
Time on Task 


9 


.767 


VI 


Frequent Monitoring of Student 
Progress 


9 


.756 


VII 


Home-School Relations 


li 


.861 




TOTAL SURVEY 


69 


.725 



^Estimated reliabilities for 1 rater are based upon the data from 
26 raters- 
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Although the continua were provided for the interviewer's 
coding only, it was still necessary to develop relatively 
discrete descriptive categories within each continuum. The 
descriptors were developed to approximate an equal interval 
scale representing a range of responses, A card sort proce- 
dure was used to improve this dimension of the interview 
coding form. Judges were given 67 envelopes with an item 
stem printed on the outside. In the envelope were five 
cards with an individual descriptor on each oije. Judges 
ordered the descriptors in a left to right sequence. The 
far right descriptor was to represent the optimal response. 
The results from the procedure were used to make further 
revisions. A second inter-rater reliability study will be 
conducted with five trained raters. Judges will listen to 
and rate a new tape containing the Interview questions and 
typical responses. Based upon the new inter-rater agreement 
and reliability indices, it will be possible to estimate the 
reliability of the interview process when employed by one 
rater in future school assessments (NEREX Report, 1981). 
The revised Interview (January, 1982) schedule contains 67 
modified-fixed alternative items and three open-ended questions. 

As data is collected using the Questionnaire in several 
schools during the spring, several psychometric analyses will 
be employed. For example, a reliability measure of internal 
consistency will be used to check the internal consistency 
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of teacher's responses within the seven scales on the form. 
These data will also provide evidence of homogeneity of the 
items within each scale. A measure of stability reliability 
will be calculated through test-retest procedures. Also, 
an exploratory factor analysis will be carried out when data 
has been collected fcr approximately 700 teachers. This 
analysis will supplement the other reliability measures and 
assist in understanding the meaningfulness of the seven 
scales on the form. 

Each component of the Connecticut School Effectiveness 
Assessment Process has been piloted and revised and is now 
being used in the form described (the Interview, Questionnaire, 
Archival Profile and Achievement Profile). More controlled 
studies designed to further develop the instruments and 
improve the process will continue. 

The Double Track: School Improvement and Research Considerations 

The primary goals of the assessment process hav^been, 
and will continue to be, addressed through a cooperative, 
planned research and development effort. Many of the proce- 
dures and activities described above have moved the assessment 
process in the direction of both goals simultaneously; although, 
certain considerations $tnd activities are more directly in 
line with one focus or the other. 

School Improvement: Data Presentation . Once multiple 
data are collected in a school they must be portrayed for 
the school faculty and principal in an integrated, coherent 

ERLC 3 1 
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framework. The data must be easily interpretable and 
sufficiently detailed so as to provide a meaningful basis 
for school decision-making. There have been a number of 
formats used in the past to accomplish this task. The 
revised assessment instruments and process have provided 
the opportunity for a reconsideration of this key aspect 
of the process. Recommendations from the NEREX Report 
identify the following questions and guidelines that were 
used in developing the format to be presented: 

1. How does this school staff perceive their own school / 
in terms of the seven characteristics of effective 
schools? " 

2. How much variability exists in these perceptions 
for each of the seven characteristics? 

3. Within each of the seven characteristics, how does 
the staff perceive specific aspeoits of their school? 
And what is the variability in these specific 
perceptions? 

4. Are these perceptions as measured by the survey and 
as measured by the questionnaire similar? If not, 
where are the differences? 

5* To what degree and in what areas are staff perceptions 

consistent with available archival data? 
6. What is the dominant achievement profile in this 

school? To what degree is this profile the same for 

low vs. middle SES students, various subject matters, 

and different grade levels? 
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Additional guidelines used in developing the data pre- 
sentation format included the following: 

1. The data presented to schools should encourage and 
facilitate positive discussion and action planning 
among school staffs. The presentation, therefore, 
should oe clear, readable, and succinct, but with 
sufficient detail that specific actions or alterna- 
tives can be discussed. 

2. The data presentation should NOT in any way attempt 

to characterize people (e.g., "50 percent of the staff 
perceives high teacher expectations, while the other 
half perceive low teacher expectations"). This could 
only promote divisiveness within a school staff. 

3. The data presentation should be descriptive, not 
prescriptive. 

4. The data presentation should emphasize substantive, 
rather than quantitative information. (In addition, , 
until adequate reliability and validity for the 
instruments have been established, the data presenta- 
tion should focus on frequencies and distributions 
rather than means and standard deviations. This 
focus might be retained even after instrument quality 
has been established.) 

5. A system for establishing a computerized data base 
for the school effectiveness project also needs to be 
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developed at this time. This system should (a) allow 
for investigations of instrument quality, (b) facilitate 
the aggregation of data for presentation to schools, 
(c) allow s for future research investigations, and (d) 
be easily modifiable since both instruments and data 
aggregation procedures are subject to change. 

(NEREX, 1981) 

A format has been developed for integrating and portray- 
ing data (see Appendix D). The format summarized item and 
scale level data from the Interview and Questionnaire responses 
collected from a school faculty* Archival data sources are 
listed in the right hand column. This format will be piloted 
in schools during the spring 1982 phase of the project. 

Assessment team leaders and local school district personnel 
have critiqued the format based on past experience. There 
was general agreement that this format would present enough 
substantive detail for informed action planning and may lead 
to further specific data gathering by school planning teams. 

Research Considerations . The evidence synthesized from 
the school and classroom effectiveness research that, sub- 
stantiates a set of school characteristics that are coincident 
with patterns of high achievement (above a locally determined 
criterion score) for all students (Brookover and Lezotte, 
1979) forms the foundation of the Connecticut School 
Effectiveness Model. Cohen (1980) and others have strongly 
recommended directions for future research in this area. An 
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analysis of how these characteristics operate in schools 
to bring about their presumed effects is not possible until 
the characteristics are operationally defined and measured. 
Construct validation studies have been designed to be a 
part of data collection and analyses in Connecticut. 

The exploratory factor analysis that has been described 
may help determine the number and nature of the underlying 
variables that are being assessed. Correlations derived 
from the Questionnaire and the Interview will be used to 
develop mul.^trait-multimethod matrices to further explore 
the hypothesized constructs. 

As more schools enter the assessment process a more 
sophisticated study can be designed to study the variance 
of student achievement scores explained by each of the 
identified school effects constructs. 

The Connecticut School Effectiveness Process attempts 
to integrate the need for research aimed at developing a 
theoretical rationale for the school effectiveness findings 
and the need for focused, school-based applications of the 
existing research in order to positively effect student 
achievement in Connecticut's schools. Efforts in this area- 
continue to develop and evolve. Research and implementation 
efforts are significantly grounded in the experiences of 
schools participating in the Connecticut project. 
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Connecticut School Effectiveness Interview 
SAMPLE ITEMS 



SAFE AND ORDERLY E NVIRONMENT 

There is an orderly, purposeful atmosphere which is free from the threat of physical harm. However, the atmosphere is 
not oppressiye and is conducive to teaching and learning. 



Is this school a safe and secure place to work? 



The school is not safe 
and secure. Fear and 
concern for physical 
safety are present. 



There is a general 
feeling of insecurity. 
It is not safe to be 
alone in the build- 
ing and numerous in- 
cidents occur. 



The school is secure 
from outside inter- 
ference. There are 
occasional incidents 
that heighten con- 
cern throughout the 
building. 



There are some 
internal student 
related problems. 
However, adults 
and students gen- 
erally feel secure, 



This is a secure 
building. Students 
and staff do not 
view security as 
an issue. 



CLEAR SCHOOL MISSION 



Thera is a clearly-articulated mission for the school through which the staff shares an understanding of and a commitment 
to instructional goals, priorities, assessment procedures and accountability. 



Is there a written statement of purpose for this school that guides the instructional program? 



There is no agreed 
upon, written state- 
ment of purpose. 
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A written statement 
exists, but it has 
little influence on 
the instructional 
program. 



A statement of 
purpose has been 
developed by admin- 
istration and facul- 
ty of this school 
A few general in- 
structional decisions 
are guided by this 
statement. 



APPENDIX A 



A statement 
exists and some 
school decisions 
result from it. 



The statement is 
the driving force 
behind most im- 
portant school 
decisions. 
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INSTRUCTIONAL LEADERSHIP 



The principal acts as. the instructional leader who effectively communicates the mission of the school to the staff, 
parents and Students and who understands and applies the characteristics of instructional effectiveness in the 
management of the instructional program of the school. 



Describe the process of a typical formal class^om observation in this school 



There is no typical 
pattern. The principal 
stops into the class- 
room and may follow- 
up with an informal 
note. 



The principal generally 
informs teacher of obser- 
vation. A lesson is ob- 
served and feedback in 
some form maybe given. 



The principal and 
teacher informally 
plan an observation. 
Feedback follows 
the observation, 
usually in the form 
of a post-conference. 



The principal and 
teacher plan for 
each observation. 
Post-conferences 
follow each ob- 
servation. 



The principal and 
teacher plan 
observation at a 
pre-conference. The 
observation is 
followed by a post- 
conference. 



HIGH EXPECTATIONS 



The school displays a climate of expectation i n which the staff believes and demonstrates that students can attain 
mastery of basic skills and that they (the stafr) hav? the capability to help students achieve such mastery. 



What do teachers in thVs schu, . believe is their responsibility in relation to student achievement in the basic skills? 



There is no specific 
responsibility. 
Teachers present 
the content. 



Tetchers are responsible 
for normal curve dis- 
tributions of achieve- 
ment according to student 
ability. 



Teachers arp re- 
sponsible fo> all 
students to master 
basic skills accord- 
ing to individual 
levels of expectancy. 



Theachers are re- 
sponsible for most 
students to master 
basic skills at 
their grade level . 



Teachers are responsibl 
for alj_ students to 
master all basic skills 
at their grade level. 



OPPORTUNITY TO LEARN AND STUDENT TIME ON TASK 



Teachers allocate a significant amount of classroom time to instruction in basic skill areas. For a high percentage of 
O ;hat allocated time students are engaged 1n planned learning activities. A? 

Rir : : ±fi 



Describe how time allotments in basic skill areas are determined in this school. 



Individual teachers 
determine their 
own schedules. 



General guidelines are 
handed down by the 
administration. 
Teachers develop 
schedules in 
partial compliance. 



The principal develops The principal and 
a general schedule. teachers agree 
Recommended time on allocated 

allotments are generally times. Schedules 
followed. are reviewed > 

monitored and/or 
adjusted if 
necessary. 



Allocated time 
in each basic skill 
areas is set with or 
by the principal . 
Teachers and 
principal value 
and monitor these 
time allotments. 



FREQUENT MONITORING OF STUDENT PROGRESS 

Feedback on student academic progress is frequently obtained. Multiple assessment methods such as teacher-made tests, 
samples of students 1 work, mastery skills checklists, criterion-referenced tests and norm- referenced tests are used. 
The results of 1 testing are used to improve individual student performance and also to improve the instructional program. 



How do xoo use the information obtained from skill tests, unit tests and/or chapter tests in the basic skills in your 
classroom? 



Information is used 
primarily to give 
students grades. 



Information is used 
for gracing and making 
groups. There is 
little individual 
feedback beyond 
grades. 



Information is used 
for grading and to 
plan general class- 
room lessons. The 
information is not 
used to modify 
instruction. 



Information is 
used to plan 
lessons for class- 
room groups and to 
give general feedback. 
Instruction is some- 
what modified based on 
resul ts . 



Information is used 
to give specific 
student feedback and 
to diagnose and pre- 
scribe appropri^.j in- 
struction. 
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H OME- SCHOOL RELATIONS 



Parents understand and support the basic mission of the school and are made to feel that they have an important role 
achieving this mission. 



To what extent are parents involved in the school? 



There is very little 
involvement of any 
kind. 



There 1s occasional 
parent involvement 
at planned school 
functions and after 
specific teacher 
requests. 



Parents are in- 
volved in the 
parent organiza- 
tion, open-houses, 
and school 
programs. 



There is an ac- 
tive parent group. 
There is general 
support, but 
limited direct 
involvement from 
most parents. 



Parents are directly 
involved in support- 
ing the school program. 
Most parents are 
involved in an over- 
all home and school 
support network that 
promotes student 
achievement. 
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Connecticut School 
Effectiveness Questionnaire 

SAMPLE ITEMS 
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1. SAFE AND ORDERLY ENVIRONMENT 

Teachers, administrators and parents assume responsibility 

for discipline in this school SD D U A SA 

2. CLEAR SCHOOL MISSION 

In mathematics, written sequential objectives exist up 

through all grades . . SD D U A SA 

3. INSTRUCTIONAL LEADERSHIP 

The principal leads frequent formal discussion concerning 

instruction and student achievement. SD D U A SA 

4. HIGH EXPECTATIONS 

Ninety to one hundred percent of the students are expected 

to master all basic skills at each grade level SD D U A SA 

5. OPPORTUNITY TO LEARN AND STUDENT TIME ON TASK 

Two hours or more are allocated for reading/la*^ uage arts 

each day throughout this school SD D U A SA 

6. FREQUENT MONITORING OF STUDENT PROGRESS 

Teachers and the principal thoroughly review and analyze 

test results to plan instructional program modifications... SD D U A SA 

7. HOME SCHOOL RELATIONS 

Most parents understand and promote the schools' instruc- 
tional program SD D U A SA 
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physical plant 



Learning 
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*afe and Orderly Environment 
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Not physically 

secure 
Fear Is present 



0* 



General Insecurity 
Niuerntis Incidents 
occur 

0 



Chaotic, disorderly, 
frequent disrup- 
tions 



A degree of con- 
trol Jed order 

frequent discipline 
problems and class 
Interruptions 0 



Cannot tell 



Teachers alone 



IS 



Unpleasant; poor Generally not safe 
light, heat, paint; or rlean 
unsafe areas 
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Students not not I* • Atmosphere not con- 
vatetl or Interested duclve to learning 
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Dfsclpl Ino problems 

with small number 

of students 
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Infrequent dis- 
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GO 
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an Issue 
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Principal/adminis- 
tration 

Teachers send many 
students to office 
0 



Teaching staff and 
principal cooper- 
ate 

SO 



Students, *taff, 
administration, 
and parents 
share responsi- 
bility 35 



Generally safe and 
clear 

Plant neutral, 
doesn't Interfere 
3S 



Building clean, 
orderly, and well 
taken rare of 



Building neat, 
clean, cornfnrl- 
able, and.snurce 
of pride 

45 



Mnst students com- 
plete required tasks 

Students receptive 
but not enthusiastic 
10 



Students generally 
positive abnut 
learning 



Host students 
eaqer and enthu- 
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Positive feeling 
In school 



* Percent selecting that response. 
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In general 
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tn n0-8l 



Vandal Ism Oat a: 
(Oral) Negligible 



Sample Item Level Data Presentation for "Safe and Orderly Environment Scale on Interview 
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